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Abstract 

We introduce a new class of non-standard variable-length codes, called 
adaptive codes. This class of codes associates a variable-length codeword 
to the symbol being encoded depending on the previous symbols in the 
input data string. An efficient algorithm for constructing adaptive codes 
of order one is presented. Then, we introduce a natural generalization of 
adaptive codes, called GA codes. 

Keywords: adaptive mechanisms, compression rate, data compres- 
sion, entropy, prefix codes, variable-length codes. 



1 Introduction 

The theory of variable-length codes |5] originated in concrete problems of infor- 
mation transmission. Especially by its language theoretic branch, the field has 
produced a great number of results, most of them with multiple applications in 
engineering and computer science. Intuitively, a variable-length code is a set of 
strings such that any concatenation of these strings can be uniquely decoded. 
We introduce a new class of non-standard variable-length codes, called adaptive 
codes, which associate a variable-length codeword to the symbol being encoded 
depending on the previous symbols in the input data string. 

The paper is organized into six sections. After this introductory section, the 
definition of adaptive codes and several theoretical remarks are given in Section 
2, as well as some characterization results for adaptive codes. The main results 
of this paper are presented in Section 3, where we focus on designing an algo- 
rithm for constructing adaptive codes of order one. In Section 4, we compute the 
entropy bounds for this algorithm. A natural generalization of adaptive codes 
is presented in Section 5. Finally, the last section contains a few concluding 
remarks. 

Before ending this introductory section, let us present some useful notation 
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used throughout the paper [SJE], and then review some basic concepts. We 
denote by \S\ the cardinality of a set S; if X is a string of finite length, then \x\ 
denotes the length of x. The empty string is denoted by A. 

For an alphabet E, we denote by E* the set 1J^L E", and by E+ the set 
lX°=i E ™' where S ° is defined by {A}. Let us denote by E-" the set (JHo S * 
and by E-™ the set lj°^„ E l . Let X be a finite and nonempty subset of E + , and 
w £ E + . A decomposition of w over X is any sequence of words u±, U2, ■ ■ ■ , Uh 
with Ui £ X, 1 < i < h, such that w — u\U2 ■ ■ - Uh- A code over E is any 
nonempty set C C E + such that each word w £ E + has at most one decompo- 
sition over C. A prefix code over E is any code C over E such that no word in 
C is proper prefix of another word in C . 

2 Adaptive Codes 

In this section we introduce a new class of non-standard variable-length codes, 
called adaptive codes. These codes are based on adaptive mechanisms, that is, 
the variable-length codeword associated to the symbol being encoded depends 
on the previous symbols in the input data string. 

Definition 2.1 Let E and A be two alphabets. A function c : E x E- — > A + , 

with n > 1, is called an adaptive code of order n if its unique homomorphic 
extension c : E* — > A* given by: 

• c(A) = A, 

• c(CTl(T2 ■ ■ ■ CT m ) = C(<71, A) C(<7 2 , <7l) . . . c(<7„- 1 , f7l(T2 ■ ■ ■ 0" n _ 2 ) 

c(cr„, CTicr 2 . . . cr„_i) c(ct„+i, dio^ • ■ ■ c„) c(er„ + 2, (T2CT3 . . . cr„+i) 

c (0Vi+3i . . . <7„+2) • ■ ■ c(f7 m , Cr m _„CT m _„-)-i . . . <7 m _i), 

for all 1 cr 2 . . . er TO G E + , is infective. 

Let us take an example in order to better understand the adaptive mecha- 
nisms presented in the definition above. 

Example 2.1 Let E = {a,b}, A = {0, 1} be alphabets, and c : E x E- 2 -> A+ 

a function given by the table below. 



E\E- 2 


a 


b 


aa 


ab 


ba 


bb 


A 


a 














1 


1 





b 


1 


1 


1 


1 








1 



One can easily verify that the function c is infective, and according to Definition 
2.1, c is an adaptive code of order two. Let x = abaa £ E + . Using the definition 
above, we encode x byc(x) — c(a, A)c(b, a)c(a, ab)c(a, ba) = 0101. 

Let c : E x E- — » A + be an adaptive code of order n, n > 1. We denote 

by C c ,a 1 a- 2 ...a h the Set 

{c(a,a\a 2 ■ ■ ■ e>7i) | cr £ E}, 
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for all o\(J2 ■ ■ ■ o~h S E-™ — {A}, and by C Ci \ the set {c(er, A) | cr G E}. We write 
C , C r 1 cr 2 ...(T ft instead of C c . a - ic72 ... a - h , and Ca instead of C c ,a whenever there is no 
confusion. 

If w G E + then we denote by w(i) the i-th symbol of w. In the rest of this 
paper we denote by AC(Yi, A, n) the set 



{c : S x E- n — > A + | c is an adaptive code of order n}. 

Theorem 2.1 Let E and A &e £wo alphabets, and let c : E x E-™ — 
6e a function. If C ai a 2 ...cr h * s a prefix code, for all o\oi ■ ■ - oh G E-™, 
cG AC(E,A,n). 



A+ 
then 



Proof Let us assume that C„ 



. CTh is prefix code, for all a\U2 ...(?(,£ £-", 
but c ^ j4C(S,A,n). By Definition 2.1, the unique homomorphic extension of 
c, denoted by c, is not injective. This implies that 3 uau' ,ua'u" € E + , with 
a, o' e E and u, u', u" G E*, such that a ^ a' and 

(*) c(uau') —c(ua'u"). 

We can rewrite (*) by 

(**) c(u)c(a, P n {u))c{u') = c(u)c(a',P n (u))c(u"), 
where P n {u) is given by 

{A if u = A, 

Mi . . . u q if u = uiu 2 ■ ■ -u q and u\, u 2 , ■ ■ ■ , u q G E and q < n, 

Uq-n+i ■ ■ - u q if u = u\u 2 . . . u q and ui, U2, ■ ■ ■ , Uq G E and q > n. 

By hypothesis, Cp n ( u ) is a prefix code and c(cr, P n (u)), c(a' , P n {u)) G Cp^). 
Therefore, the set {c(a, P n (u)), c(a' , P n (u))} is a prefix code. But the equality 
(**) can hold if and only if {c(ct, P n (u)), c(a' , P n (u))} is not a prefix set. Hence, 
our assumption leads to a contradiction. ^> 

Remark 2.1 The converse of Theorem 2.1 does not hold. We can prove this 
by taking a counter-example. Let us consider E = {a, b} and A = {0,1} two 
alphabets, and c:ExS- 2 -> A + a function given by the table below. 



E\E- 2 


a 


b 


aa 


ab 


ba 


bb 


A 


a 























b 


01 


1 


1 


1 


1 


1 


1 



One can verify that the unique homomorphic extension of c, denoted by c, is 
injective. Therefore, we conclude that the function c is an adaptive code of 
order two. 

Let E, A, and Bool = {True, False} be alphabets. We define the function 
Prefix : AC(E,A,n) -» Bool by: 



Prefix (c) 



True if C u is a prefix code, for all u G E- n , 
False otherwise. 
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The function Prefix can now be used to translate the hypothesis in Theorem 
2.1: if c : S x E-™ — ► A+ is a function satisfying Prefix(c) = True, then we 
conclude that c G j4C(E, A, n). 

Let c G ^4C(E, A, n) be an adaptive code satisfying Prefix(c) = True. Then, 
the algorithm Decoder described below requires a linear time. 





Decoder(c, u) 




input: c G AC(T,, A, n) smc/i t/iaf Prefix(c) = True and u G A+; 




output: w G E + suc/i iftai c(w) — u; 




begin 


1. 


u; := A; i := 1; Last := A; length := m|; 


2. 


while i < length do 




begin 


3. 


Lef cr G S 6e i/ie unique symbol of E i/ie property 




that c(a, Last) is prefix of u(i) ■ u(i + 1) ■ . . . • u(length); 


4. 


w := w ■ a; 


5. 


i := i + \c(a, Last)\; 


6. 


if |Last < n 


7. 


then Last := Last ■ cr; 


8. 


else iasi := Last(\Last\ — n + 2) • . . . • Last(\Last\) ■ a; 




end 


9. 


return w; 




end 



Remark 2.2 In the third step of the algorithm given above, the symbol denoted 
by a is unique with that property due to the input restrictions. 

Remark 2.3 One can easily verify that the while loop in algorithm Decoder 
is iterated 

h 

M ~ ^2{\c(w l ,P n (w 1 w 2 ■ ..u>i_i))| - 1) 

i=l 

times, where w — W\W 2 ■ ■ ■ Wh, and P n is the function given in Theorem 2.1. 

In practice, we can use only adaptive codes satisfying the equality Prefix (c) = 
True, since designing a decoding algorithm for the other case requires additional 
information and more complicated techniques. 

3 Data Compression using Adaptive Codes 

The construction of adaptive codes requires different approaches, depending 
on the structure of the input data strings. In this section, we focus on data 
compression using adaptive codes of order one. 
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Definition 3.1 Let E be an alphabet and w = w±w 2 ■ ■ ■ Wh G £- 2 , with Wi G E, 
for all i G {1, 2, . . . , /i}. j4 subword uu of w, with h£E, is called a pair o/ w. 

Remark 3.1 Let £ 6e an alphabet and w = w\w 2 ■ ■ ■ u>h G £- 2 . It is useful to 
consider the following notations: 

1. Pairs(w) = {i | 1 < i < \w\ — 1, Wi = Wi + i}, 

2. NRpairs(w) — \ Pairs (w)\, 



The main goal of this section is to design an algorithm for constructing 
adaptive codes of order one, under the assumption that the input data strings 
have a large number of pairs. 

Let E = {(Ti, o- 2 , ■ • ■ , o~h~} and A = {0, 1} be alphabets, c G AC(E, A, 1) an 
adaptive code of order one, and w € E + . We denote by A c the matrix given by: 



\ c(a h ,ai) c(a h ,a 2 ) ■■■ c(cr hl a h ) c(a h ,X) J 

Let us denote by Huffman(i?F(w), n) the well-known Huffman's algorithm 
0, where n > 1, and EF(w) is the matrix given below. 



We assume that the first row of the matrix EF(w) contains the symbols 
which are being encoded, while the second row contains their frequencies, that 
is, f(o~i) is the frequency of the symbol o~i in w. 

Also, we assume that Huffman(fiF (w), n) is the matrix given by 

Huffman(£F(>),n) = ( H{a ll w) H(a 2 ,w) ... H(a n ,w) ) 

where H(o~i,w) is the codeword associated to the symbol o~i by Huffman's al- 
gorithm. The algorithm Builder described further on takes linear time, and 
constructs an adaptive code of order one satisfying Prefix{c) = True. 

Proposition 3.1 Let c : S x E- 1 — > {0, 1} + be a function given by the matrix 
Builder{c). Then, c G AC(E, {0, 1}, 1) and Prefix (c) = True. 

Proof Applying the algorithm Builder to the function c, one can easily 
verify that Prefix(c) — True. Therefore, according to Theorem 2.1, c is an 
adaptive code of order one, that is, c G ^4C(E, {0, f }, 1). (} 



3. Prate(w) 




A c 



I c(<7i,<7i) c(a 1 ,a 2 ) 
c(<7 2 ,o-i) c(a 2 ,a 2 ) 



c(ai,a h ) c(cti,A) \ 
c{cr 2 ,o- h ) c(a 2l X) 
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Builder(c) 

input: c : S x S^ 1 -> {0, 1}+, E = <7 2 , . . . , <r h }; 

output: A c such that c G j4C(£, {0, 1}, 1) and Prefix(c) = True; 

begin 

1. for i := 1 to h do A c (i,z) := 0; 

o P . f CT 2 ct 3 . . . o h \ 

2 - 25 - (, ... )' 

3. X := Kuftman(E,h- 1); 

4. for i := 2 to /i do 
begin 

5. A c (l,i) :=l-X(l,i-l); 

6. 1) := 1); 

7. A c (i,l) :=X(l,i-l); 
end 

8. for j := 2 to /i do 
begin 

9. for i := 2 to j - 1 do A c (i, j) := X(l,i- 1); 

10. for i :=j + 1 to h do A c (i, j) := 1); 
end 

11. for i := 1 to h do A c (i,/i+ 1) := A c (l,i); 

12. return j4 c ; 
end 



Example 3.1 Lei c : {a, b, c} x {a, b, c}- 1 — > {0,1} + 6e a function. One can 
verify that A c is the matrix given below. 

/ 10 11 \ 
A c = 11 10 10 . 

\ io ii o ii y 

Let w = abbbcabccaabccabbcba be an input data string. It is easy to verify 
that Pairs (w) — {2, 3, 8, 10, 13, 16}, NRpairs(w) = 6, and Prate(w) = 0.3. 

Encoding the string w by c requires the computation ofc{w) . Using Definition 
2.1, we get that \c(w)\ = 33. 

Let us apply Huffman's algorithm to the data string w in order to make 
a comparison between the results. If we denote by Huffman(w) the codeword 
associated to w by Huffman's algorithm, we get that \Huf fman(w)\ — 32. An 
even better result can be obtained when the input data string has a larger number 
of pairs, as shown in the following example. 

Example 3.2 Let c : {a, b, c} x {a, b, c}- 1 — > {0, 1} + be an adaptive code of 
order one given as in the previous example, and w = abbbccbccaabccaaacba 
an input data string. One can verify that Pairs(w) — {2, 3, 5, 8, 10, 13, 15, 16}, 
NRpairs(w) = 8, Prate(w) = 0.4, and \c(w)\ = 31. Encoding the string w by 
Huffman's algorithm, we get that \Huffman{w)\ — 34. 
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The results obtained in the previous examples are summarized in the table 
below, which shows that we get substantial improvements for input data strings 
having a larger number of pairs. 



w 


NRpairs (w) 


Prate(w) 


\c(w)\ 


\Huffman(w)\ 


abbbcabccaabccabbcba 


6 


0.3 


33 


32 


abbbccbccaabccaaacba 


8 


0.4 


31 


34 



4 Builder: Entropy Bounds 

In this section, we focus on computing the entropy bounds for the algorithm de- 
scribed in section 3. Given that our algorithm is based on Huffman's algorithm, 
let us first recall the entropy bounds for Huffman codes. 

Definition 4.1 Let S be an alphabet, x a data string of length n over £ and 
k the length of the encoder output, when the input is x. The compression rate, 
denoted by R{x), is defined by 

R(x) = -. 
n 

Let R(x) be the compression rate in codebits per datasample, computed after 
encoding the data string x by the Huffman algorithm. One can obtain upper 
and lower bounds on R{x) before encoding the data string x by computing the 
entropy denoted by H(x). Let x be a data string of length n, (Fi,F 2 , . . . , Fh) 
the vector of frequencies of the symbols in x and k the length of the encoder 
output. The entropy H(x) of x is defined by 

Let Li be the length of the codeword associated to the symbol with the 
frequency Fi by the Huffman algorithm, 1 < i < h. Then, the compression rate 
R(x) can be re-written by 

h 

R(x) = -Y / F i L i . 
n z — ' 

z— 1 

If we relate the entropy H{x) to the compression rate R{x), we obtain the 
following inequalities: 

H(x) < R(x) < H(x) + 1. 

Let S = {<7i, cr 2 , . . . , (7i} be an alphabet and c : X x S- 1 — » {0, 1} + an 
adaptive code of order one constructed as shown in section 3. Also, consider 
w = wiw 2 ■ ■ ■ w s G £+, Wi e S, 1 < i < s, and p the number of symbols 
occurring in w. 

We denote by Ra (w) the compression rate obtained when encoding the string 
w by c and by Ha(w) the entropy of w. It is useful to consider the following 
notations: 
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1. EH(w) = {i | 2 < i < s and w, ^ u^-i}, 

2. LNotHuffman(w) = \c(w u A)| + E ie p a ir S ( w ) |cK+i,w»)|, 

3. LHuffman(w) is the entropy of Wj 1 Wj 2 . . . Wj r , jk 6 EH(w), 1 < k < r, 

4. Ha(w) = LNotHuffman{w) + LHuffman(w). 

It is easy to verify that LNotHuffman(w) = NRPairs(w) + \c(wi, A)|. Using 
the notation above, we get that 

LHuffman{w) = £ {-^ ]T [^(^(l + log 2 ^^)]}, 

ieEH(w) ^ l) q£Prev(w,) q( l) 

where 

• 7V( Wi ) = |{j | j e £ff(u;) and = 

• Prev(wi) = {j \ j + lG EH(w) and Wj + i — Wi}, 

• F q (wi) — \{j | j G EH(w) and = and Wj-i = g E Prev(wi). 

Finally we can relate the entropy Ha (w) to the compression rate Ra (w) by 
the following inequalities: 

H A {w) < R A (w) < H A (w) + 1, 

where Ra(w) is given by 

|c(wi,A)c(u;2,wi)...c(u; s ,w s _i)| 
s 

5 GA Codes 

In this section, we introduce a natural generalization of adaptive codes (of any 
order), called GA codes (Generalized Adaptive codes). Theorem 5.1 proves 
that adaptive codes are particular cases of GA codes. 

Definition 5.1 Let S and A be two alphabets and F : N* x S+ — > S* a 

function, where N is the set of natural numbers, and N* — N — {0}. A function 
cf '■ £ x S* — > A + is called a GA code z/ its unique homomorphic extension 
W : S* — > A* gwen fry 

• cF(A) = A, 

• C^(criCT 2 . . . (T m ) = C F (ai,F( 1,(710-2 . . . <7 m )) . . . Cp (<j m , F(m, <Ti<T 2 • • - o- m )), 
for all o~ io~2 ' • • o~m E S + , is injective. 

Remark 5.1 The function F in Definition 5.1 is called the adaptive function 
of the GA code cp. Clearly, a GA code cp can be constructed if its adaptive 
function F is already constructed. 
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Remark 5.2 Let S and A be two alphabets. We denote by GAC(T,, A) the set 
{c F : £ x S* -> A+ | c F is a GL4 code}. 

Theorem 5.1 Let £ and A 6e alphabets. Then, AC(T,,A,n) C CL4C(£,A), 
/or all n > 1 . 

Proof Let cp G AC(E, A, n) be an adaptive code of order n, n > 1, and 
F : JV* x S + ^ S* a function given by: 



for alii > 1 and <TiO"2 . . . cr m G E + . One can verify that \F(i, o~\o~2 ■ ■ ■ o~ m )\ < n, 
for all i > 1 and o\Oi . . . a m G S + . According to Definition 2.1, the function cf 
is given by: 



for all (Tio-2 ... cr m G S + . It is easy to remark that 

• C^(criCT2 . . . CT m ) = Cf(cTi,F(1, O-1CT2 • ■ • CT m )) . . ■ Cp(o~ mi F(jn, 0~\0~2 ■ ■ - CF m )), 

for all C1CT2 • ■ ■ 0"m S S+, which proves the theorem. 

6 Conclusions and Future Work 

We introduced a new class of non-standard variable- length codes, called adaptive 
codes, which associate a variable-length codeword to the symbol being encoded 
depending on the previous symbols in the input data string. The main results 
of this paper are presented in Section 3, where we have shown that if an input 
data string x has a significant number of pairs, then a good compression rate is 
achieved when encoding x by adaptive codes of order one. 

In a further paper devoted to adaptive codes, we intend to extend the algo- 
rithm Builder to adaptive codes of any order. 
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